Skip to content

feat(driver): add Cloudflare Image Bed support#2427

Open
ZZ0YY wants to merge 26 commits into
OpenListTeam:mainfrom
ZZ0YY:feat/cfimgbed
Open

feat(driver): add Cloudflare Image Bed support#2427
ZZ0YY wants to merge 26 commits into
OpenListTeam:mainfrom
ZZ0YY:feat/cfimgbed

Conversation

@ZZ0YY
Copy link
Copy Markdown

@ZZ0YY ZZ0YY commented May 1, 2026 •
edited
Loading

Description / 描述

目前实现了列表和下载功能，上传功能后续再做，已完成列表，下载，上传，上传时自动创建文件夹先提一个 PR，看看意见

Add a new storage driver for Cloudflare Image Bed https://github.com/MarSeventh/CloudFlare-ImgBed
(a popular image hosting solution based on Cloudflare Workers & KV/D1).
Features implemented:

  • List: Support for file and directory listing with custom RootPath (subdirectory) mounting.
  • Link: Direct download/view links for images and files.
  • Metadata: Correctly parses file size, modification time (from timestamp), and MIME types from API metadata.

为 Cloudflare Image Bed
https://github.com/MarSeventh/CloudFlare-ImgBed
（基于 Cloudflare Workers 和 KV/D1 的图床方案）新增存储驱动。
实现功能：

  • 列表 (List)：支持文件和目录列表，支持自定义根目录（子目录）挂载。
  • 直链 (Link)：支持获取图片的直链下载/预览地址。
  • 元数据 (Metadata)：正确解析 API 返回的文件大小、修改时间（时间戳转换）及 MIME 类型。

Motivation and Context / 背景

Cloudflare Image Bed is a lightweight and free-tier-friendly image hosting tool. Integrating it into OpenList allows users to manage their Cloudflare-based image assets alongside other storage services.

Cloudflare Image Bed 是一个轻量且对免费用户友好的图床工具。通过将其集成到 OpenList，用户可以方便地与其他存储服务一起管理其在 Cloudflare 上的图片资产。

Relates to #376 (Add more storage drivers)

How Has This Been Tested? / 测试

  • Environment: Go 1.24, Windows 10.

  • Manual Test:

    1. Mounted a real CFImgBed instance with Token.
    2. Verified that all files and folders are listed correctly in the dashboard.
    3. Verified that files in subdirectories are accessible when using a specific RootPath.
    4. Verified that images can be previewed/downloaded via the generated links.

  • Debug: Used resty debug mode to ensure HTTP requests/responses are handled correctly.

  • 环境：Go 1.24, Windows 10。

  • 手动测试：

    1. 使用真实的 CFImgBed 实例和 Token 进行了挂载。
    2. 验证了所有文件和文件夹在面板中正确列出。
    3. 验证了设置“根目录路径”后，子目录下的文件访问正常。
    4. 验证了生成的图片直链可以正常预览和下载。

  • 调试：开启 resty 的调试模式，确保 HTTP 请求和响应逻辑无误。

Checklist / 检查清单

  • I have read the CONTRIBUTING document.
    我已阅读 CONTRIBUTING 文档。
  • I have formatted my code with go fmt.
    我已使用 go fmt 格式化提交的代码。
  • I have added appropriate labels to this PR.
    我已为此 PR 添加了适当的标签。
  • I have updated the repository accordingly.
    我已相应更新了相关仓库。

@ZZ0YY ZZ0YY mentioned this pull request May 1, 2026
Closed
9 tasks
jyxjjj
jyxjjj requested changes May 1, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@jyxjjj jyxjjj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

驱动名称 还有别瞎删文件 Issue 也不是给你找审核用的

@jyxjjj
Copy link
Copy Markdown
Member

jyxjjj commented May 1, 2026

可能违反使用政策

@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 1, 2026 •
edited
Loading

驱动名称 还有别瞎删文件 Issue 也不是给你找审核用的

不好意思 不熟悉流程，急于询问相关要求 所以提了个 issue。入门试一试，前端构建文件忘记弄回去了

@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 1, 2026 •
edited
Loading

可能违反使用政策

应该没有吧，单纯看使用者使用什么存储后端，比如我自己是用 S3和 Cloudflare 自己的对象存储，完全合规。较少使用Huface 和 telegram 存储文件
我写这个主要是为了方便上传

ZZ0YY and others added 2 commits May 1, 2026 14:00
@ZZ0YY
fix: restore accidentally deleted file and name
9bdaac8
@ZZ0YY
refactor: rename driver to cloudflare_imgbed and fix module structure
f828fc5 
- Rename driver identifier and directory to 'cloudflare_imgbed' for consistency.
- Remove invalid 'replace' directive in go.mod.
- Restore accidentally modified/deleted files in public/dist.
- Update driver registration in drivers/all.go.

Co-authored-by: Copilot <copilot@github.com>
@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 1, 2026

抱歉，我是第一次为Openlist贡献代码，对命名规范和 Module 处理不熟悉 已经按照建议更正了驱动名称（改为 cloudflare_imgbed），并移除了 go.mod 中的 replace 逻辑，已经更正了，请有空再指教
屏幕截图 2026-05-01 105822

顺便补个运行截图
屏幕截图 2026-05-01 105803

@ZZ0YY ZZ0YY requested a review from jyxjjj May 1, 2026 06:08
xrgzs
xrgzs reviewed May 1, 2026
View reviewed changes
Comment thread drivers/cloudflare_imgbed/driver.go Outdated
xrgzs
xrgzs reviewed May 1, 2026
View reviewed changes
Comment thread drivers/cloudflare_imgbed/meta.go Outdated
ZZ0YY and others added 3 commits May 1, 2026 17:48
@ZZ0YY
fix: use base.NewRestyClient() and use e.g
7858f49 
Co-authored-by: Copilot <copilot@github.com>
@ZZ0YY
fix:go fmt
569dedf
@ZZ0YY
feat(driver/cloudflare-imgbed): enhance cloudflare_imgbed API integra…
36aecbf 
…tion with improved error handling and pagination
@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 1, 2026 •
edited
Loading

@xrgzs 您好 新的提交进行了
1：错误处理优化：增强了对 API 返回异常的捕获与处理，提高稳定性。
2：分页支持：新增了分页功能，有效应对大数据量返回时的性能消耗。
其背景是图片往往有上千张上万张，之前的写法可能性能此时会不好。本人一个目录下3240张图片经过测试没啥问题，虽然可能不太专业
3：使用utils.EncodePath(fullPath) 进行安全编码

并且使用了中文注释，进行了分页测试，没有问题 您可以直接查看这个commit 36aecbf

以下是截图
屏幕截图 2026-05-02 015103

@ZZ0YY ZZ0YY force-pushed the feat/cfimgbed branch from 8ee3f67 to 36aecbf Compare May 1, 2026 23:25
@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 1, 2026

不好意思 刚刚在调试 build 工作流 拉 snap 老是不成功

@xrgzs
Copy link
Copy Markdown
Member

xrgzs commented May 2, 2026

有webdav https://cfbed.sanyue.de/api/webdav.html

@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 2, 2026 •
edited
Loading

有webdav https://cfbed.sanyue.de/api/webdav.html

已考虑和测试，无法实现大于 100 MB 的文件上传，受限于 cloud flare，故才来提交这个新驱动

而且其实在 AI 的帮助下，上传逻辑也写好了，不过感觉测试还不够充分，我只简单地测试了一下，上传大于 100mb 的文件成功了，目前是处于能用的阶段， 但是我感觉写法不够完善，担心可能增加您审核的负担，三天后 我又要开学了🥲
在考虑要不要提另一个 PR？望请指导在此大型项目中我应该如何贡献呢？

@j2rong4cn j2rong4cn marked this pull request as draft May 3, 2026 02:40
j2rong4cn
j2rong4cn reviewed May 3, 2026
View reviewed changes
Comment thread drivers/cloudflare_imgbed/driver.go
@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 3, 2026 •
edited
Loading

@j2rong4cn 您好 看到您标记为草稿我以为我写得不好来着 已经在改了 刚才在搞上传相关的逻辑，这里也有一些变动，避免重复修改，推荐您等我写好完整的再一起看吧，应该快完成了

@j2rong4cn
Copy link
Copy Markdown
Member

j2rong4cn commented May 3, 2026

分片上传参考可以这个，stream.NewStreamSectionReader + errgroup.NewOrderedGroupWithContext

OpenList/drivers/123_open/upload.go

Lines 45 to 172 in 7e37c40

func (d *Open123) Upload(ctx context.Context, file model.FileStreamer, createResp *UploadCreateResp, up driver.UpdateProgress) error {
uploadDomain := createResp.Data.Servers[0]
size := file.GetSize()
chunkSize := createResp.Data.SliceSize
ss, err := stream.NewStreamSectionReader(file, int(chunkSize), &up)
if err != nil {
return err
}
uploadNums := (size + chunkSize - 1) / chunkSize
thread := min(int(uploadNums), d.UploadThread)
threadG, uploadCtx := errgroup.NewOrderedGroupWithContext(ctx, thread,
retry.Attempts(3),
retry.Delay(time.Second),
retry.DelayType(retry.BackOffDelay))
for partIndex := range uploadNums {
if utils.IsCanceled(uploadCtx) {
break
}
partIndex := partIndex
partNumber := partIndex + 1 // 分片号从1开始
offset := partIndex * chunkSize
size := min(chunkSize, size-offset)
var reader io.ReadSeeker
var rateLimitedRd io.Reader
sliceMD5 := ""
// 表单
b := bytes.NewBuffer(make([]byte, 0, 2048))
threadG.GoWithLifecycle(errgroup.Lifecycle{
Before: func(ctx context.Context) (err error) {
reader, err = ss.GetSectionReader(offset, size)
return
},
Do: func(ctx context.Context) (err error) {
reader.Seek(0, io.SeekStart)
if sliceMD5 == "" {
// 把耗时的计算放在这里，避免阻塞其他协程
sliceMD5, err = utils.HashReader(utils.MD5, reader)
if err != nil {
return err
}
reader.Seek(0, io.SeekStart)
}
b.Reset()
w := multipart.NewWriter(b)
// 添加表单字段
err = w.WriteField("preuploadID", createResp.Data.PreuploadID)
if err != nil {
return err
}
err = w.WriteField("sliceNo", strconv.FormatInt(partNumber, 10))
if err != nil {
return err
}
err = w.WriteField("sliceMD5", sliceMD5)
if err != nil {
return err
}
// 写入文件内容
_, err = w.CreateFormFile("slice", fmt.Sprintf("%s.part%d", file.GetName(), partNumber))
if err != nil {
return err
}
headSize := b.Len()
err = w.Close()
if err != nil {
return err
}
head := bytes.NewReader(b.Bytes()[:headSize])
tail := bytes.NewReader(b.Bytes()[headSize:])
rateLimitedRd = driver.NewLimitedUploadStream(ctx, io.MultiReader(head, reader, tail))
token, err := d.getAccessToken(false)
if err != nil {
return err
}
// 创建请求并设置header
req, err := http.NewRequestWithContext(ctx, http.MethodPost, uploadDomain+"/upload/v2/file/slice", rateLimitedRd)
if err != nil {
return err
}
// 设置请求头
req.Header.Add("Authorization", "Bearer "+token)
req.Header.Add("Content-Type", w.FormDataContentType())
req.Header.Add("Platform", "open_platform")
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
if res.StatusCode != 200 {
return fmt.Errorf("slice %d upload failed, status code: %d", partNumber, res.StatusCode)
}
b.Reset()
_, err = b.ReadFrom(res.Body)
if err != nil {
return err
}
var resp BaseResp
err = json.Unmarshal(b.Bytes(), &resp)
if err != nil {
return err
}
if resp.Code != 0 {
return fmt.Errorf("slice %d upload failed: %s", partNumber, resp.Message)
}
progress := 100 * float64(threadG.Success()+1) / float64(uploadNums+1)
up(progress)
return nil
},
After: func(err error) {
ss.FreeSectionReader(reader)
},
})
}
if err := threadG.Wait(); err != nil {
return err
}
return nil
}

@ZZ0YY
feat(cloudflare_imgbed): implement upload functionality and optimize …
a54f30b 
…performance

- Added support for standard multipart form upload with zero-copy streaming.
- Implemented HuggingFace LFS direct upload for large files (>20MB).
- Integrated with OpenList global rate limiter and progress tracking.
- Optimized memory usage using io.MultiReader for request body construction.
- Added configurable upload threads for chunked HF uploads.
- Support auto mkdir dir when in upload
@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 3, 2026

@j2rong4cn 您好 已经完成上传模块编写并且测试完成，正是学习123open。
屏幕截图 2026-05-03 172208

屏幕截图 2026-05-03 170509

@j2rong4cn
Copy link
Copy Markdown
Member

j2rong4cn commented May 3, 2026

路径处理其实很简单的，你看着改吧dc74222

@j2rong4cn
Copy link
Copy Markdown
Member

j2rong4cn commented May 3, 2026

Obj的ID和Path只在当前驱动使用，List返回什么值，Put、Link等方法就会收到什么

@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 3, 2026

Obj的ID和Path只在当前驱动使用，List返回什么值，Put、Link等方法就会收到什么

感谢指导，之前吃了亏。现已精简逻辑，确实可运行

@j2rong4cn j2rong4cn marked this pull request as ready for review May 3, 2026 13:40
@j2rong4cn
Copy link
Copy Markdown
Member

j2rong4cn commented May 3, 2026

我没测试

@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 3, 2026 •
edited
Loading

我没测试

这里是用 Windows 10 amd 64 环境测的
稍微再次测了一下自动建文件夹功能没有问题
Screenshot_2026-05-03-22-04-26-65_df198e732186825c8df26e3c5a10d7cd.jpg

Screenshot_2026-05-03-22-03-30-44_df198e732186825c8df26e3c5a10d7cd.jpg

而且能够自动分小文件和大文件上传，如图 ：Screenshot_2026-05-03-22-10-30-18_df198e732186825c8df26e3c5a10d7cd.jpg

HF 是 haface 渠道，TG 就是 telegram
还有就是视频播放正常 图片打开正常
Screenshot_2026-05-03-22-09-04-40_df198e732186825c8df26e3c5a10d7cd.jpg

Screenshot_2026-05-03-22-08-33-44_df198e732186825c8df26e3c5a10d7cd.jpg

目前在跑批量任务 10 个 g 的压缩包
测试上传稳定性
Screenshot_2026-05-03-22-07-31-63_df198e732186825c8df26e3c5a10d7cd.jpg

截止 5 月 3 号22:48

稳定性还可以，前提是要限制上传数，可以写文档提示

bug：多级文件夹上传就会出问题，由于官方 API 未提供创建文件夹的接口，本来这个创建文件夹逻辑就是直接通过上传时通过上传接口携带上传路径，就可以实现建立文件夹，后端会自动处理新建文件夹的逻辑，就成功了，但是当此种情况，就会失败：failed to get dir [/test/test/test2]: object not found
如图：我把本地存储的
/test
├── 文件1
├── 文件2
└── test2
├── 文件3
└── 文件4
这种文件夹上传，文件 1 和文件 2 成功了 文件 3 和文件 4 失败了
Screenshot_2026-05-03-22-47-29-01_df198e732186825c8df26e3c5a10d7cd.jpg

应该是由于复制任务的逻辑是先创建文件夹，再把文件放进去，一层文件夹没有问题 但是当有两层的时候，第二层文件夹就会实际上创建失败？虽然感觉分析还不够合理，但是目前只能先做到这了

同时，如果复制空文件夹，就会导致 openlist 这里显示了，但实际上后端根本没有创建

感谢您的指导！我此次五一假期的开发任务只能先进行到这了！

@ZZ0YY ZZ0YY mentioned this pull request May 4, 2026
Closed
1 task
j2rong4cn and others added 5 commits May 27, 2026 14:33
@j2rong4cn
Merge branch 'main' into feat/cfimgbed
57c8cd0
@j2rong4cn
feat(cloudflare_imgbed): add virtual directory support
89bd84c
@j2rong4cn
Merge branch 'main' into feat/cfimgbed
bd3c66a
@j2rong4cn
refactor(weak_cache): iImprove weak pointer cleanup handling
fd59e29 
Store a cleanup handle alongside weak pointers and stop previous cleanups when overwriting entries to avoid stale removals and leaked cleanup handlers.

Ensure Delete signals success and stops associated cleanup. Add Clear to stop all active cleanups. Use entry identity in the cleanup callback to avoid removing newly inserted values.
@j2rong4cn
fix bug
50dd55d
@j2rong4cn j2rong4cn requested a review from Copilot May 28, 2026 07:42
Copilot AI reviewed May 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new storage driver integrating Cloudflare Image Bed (Cloudflare Workers-based image hosting) into OpenList, including support for listing, direct links, and uploads (with optional HuggingFace LFS direct upload for large files). It also introduces an internal weak-reference cache utility to manage virtual directories without manual cleanup goroutines.

Changes:

  • Added cloudflare_imgbed driver with List/Link/Remove/MakeDir/Put (standard multipart upload + HuggingFace direct/chunked upload).
  • Added a generic WeakCacheMap (weak refs + runtime.AddCleanup) to store ephemeral virtual directory objects.
  • Registered the new driver in the global drivers list.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
internal/cache/weak_cache.go Adds a generic weak-reference cache map used to track virtual objects with GC-based cleanup.
drivers/cloudflare_imgbed/driver.go Implements core driver methods (Init/Drop/List/Link/Get/MakeDir/Remove) and virtual dir handling.
drivers/cloudflare_imgbed/meta.go Defines driver addition/config metadata and registers the driver.
drivers/cloudflare_imgbed/types.go Adds API response types and metadata parsing helpers for listing.
drivers/cloudflare_imgbed/util.go Adds a shared Resty request helper with retries and API error parsing.
drivers/cloudflare_imgbed/upload.go Implements standard multipart upload and HuggingFace direct (incl. chunked) upload + commit.
drivers/all.go Registers the new cloudflare_imgbed driver import.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread drivers/cloudflare_imgbed/util.go
Comment thread drivers/cloudflare_imgbed/util.go Outdated
Comment thread drivers/cloudflare_imgbed/upload.go Outdated
Comment thread drivers/cloudflare_imgbed/upload.go
Comment thread drivers/cloudflare_imgbed/upload.go
Comment thread drivers/cloudflare_imgbed/upload.go Outdated
Comment thread drivers/cloudflare_imgbed/upload.go Outdated
Comment thread drivers/cloudflare_imgbed/driver.go
j2rong4cn and others added 4 commits May 28, 2026 15:57
@j2rong4cn
Apply suggestions from code review
488454a 
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
@j2rong4cn
Adds uploadFolder param and streams base64 sample
e182152 
Adds an uploadFolder query parameter to forward the destination path to the backend. Replaces the fixed-size sample read with a streaming base64 encoder to avoid truncation and reduce allocations

Co-authored-by: Copilot <copilot@github.com>
@j2rong4cn
refactor: add context parameter to doRequest and related calls
8013801
@j2rong4cn
fix: update List method to check args.Refresh before virtual director…
a73075a 
…y mask
@j2rong4cn
Copy link
Copy Markdown
Member

j2rong4cn commented May 28, 2026

@ZZ0YY
新增虚拟文件夹支持，只在真实文件夹缓存有效期内显示，刷新缓存后不显示虚拟文件夹，但可以通过地址访问虚拟文件夹
大白话：二级虚拟文件夹为隐藏目录，列表不显示，但可以访问

@ZZ0YY
Copy link
Copy Markdown
Author

ZZ0YY commented May 29, 2026

@ZZ0YY
新增虚拟文件夹支持，只在真实文件夹缓存有效期内显示，刷新缓存后不显示虚拟文件夹，但可以通过地址访问虚拟文件夹
大白话：二级虚拟文件夹为隐藏目录，列表不显示，但可以访问

太谢谢您了 我在备考高考 要鸽到考完再继续了｡°o

@ZZ0YY ZZ0YY mentioned this pull request May 29, 2026
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xrgzs xrgzs xrgzs left review comments

@j2rong4cn j2rong4cn j2rong4cn left review comments

Copilot code review Copilot Copilot left review comments

@jyxjjj jyxjjj Awaiting requested review from jyxjjj

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ZZ0YY @jyxjjj @xrgzs @j2rong4cn